mongo: partition snapshots for numeric and string _id collections by andreyzhelnin-st · Pull Request #4487 · PeerDB-io/peerdb

andreyzhelnin-st · 2026-06-24T15:40:34Z

Summary

Extends the MongoDB connector so initial snapshots parallelize for numeric and string _id collections, not just ObjectID. Previously any non-ObjectID _id fell back to a single serial full-table partition, making large naturally-keyed collections (app-store numeric IDs, package-name strings) too slow to snapshot inside the oplog retention window.

Based on v0.36.29 (our deployed release).
It improves initial sync (MongoDb -> Clickhouse from 10 hours to 3 hours on 8mln collection).

Approach

GetQRepPartitions dispatches on the collection's min/max _id BSON type:

ObjectID — unchanged (uniform division of the 12-byte keyspace).
Int32/Int64 — uniform min/max division, reusing utils.PartitionHelper to emit non-overlapping IntPartitionRange partitions ($gte/$lte).
String — sampling-based quantile boundaries via $sample (honours read preference), anchored by the real min/max, emitted as a new StringPartitionRange proto type with contiguous half-open ranges ($gte/$lt).
Mixed-type / Double / empty — safe full-table fallback.

Changes

protos/flow.proto — add StringPartitionRange + string_range oneof field.
connectors/mongo/qrep_partition.go — type dispatcher; numeric + string builders; pure, unit-tested computeStringBoundaries.
connectors/mongo/qrep.go — IntRange/StringRange cases in toRangeFilter.
connectors/utils/monitoring/monitoring.go — handle StringRange in addPartitionToQRepRun (avoids the unknown range type error path).

Testing

go build ./..., go vet clean.
New unit tests for computeStringBoundaries (even/clustered/sparse/duplicate/out-of-range/empty + contiguity), toRangeFilter (Int/String/ObjectID/unsupported), and _id type detection — all passing.

Notes / limitations

Numeric uses uniform value division; monitor partition balance on first run for sparse ID spaces.
Double _id and mixed-type collections fall back to full-table by design.
Requires regenerating protobuf bindings (./generate-protos.sh) before building images, as flow/generated is gitignored.

CLAassistant · 2026-06-24T15:41:32Z

All committers have signed the CLA.

jgao54 · 2026-06-25T03:05:14Z

+message StringPartitionRange {
+  string start = 1;
+  string end = 2;
+}


we are correctly working on StringPartitionRange for MySQL and decide to have StringPartitionRange introduced an end_inclusive boolean flag for String (see #4482), bringing in a small optimization where we don't have to reserve one of the partitions for the last row/document.

jgao54 · 2026-06-25T03:25:59Z

+	case *protos.PartitionRange_IntRange:
+		// Numeric _id: inclusive range. PartitionHelper builds non-overlapping
+		// [start, end] integer ranges. Mongo compares int32/int64 numerically, so
+		// int64 bounds match int32-typed _ids.
+		return bson.D{
+			bson.E{Key: watermarkColumn, Value: bson.D{
+				bson.E{Key: "$gte", Value: r.IntRange.Start},
+				bson.E{Key: "$lte", Value: r.IntRange.End},
+			}},
+		}, nil


for mongo, using inclusive-inclusive int range [start, end] is prone to missing data in the rare scenario where user has doubles mixed in with integers. For example, if the _ids are 1, 2, 2.5, 3, 4; min/max will show up as 1 and 4, ended up with ranges [1, 2], [3, 4], we'd miss the 2.5 document.

we could introduce a NumericPartitionRange and use inclusive-exclusive [start, end), similar to StringPartitionRange, to avoid any gaps.

(While Mongo sorts by type so we don't have to worry about this for ObjectIDs, all numeric types are in the same type bucket)

jgao54

Thank you for your contribution. Using sampling for string is a great strategy for MongoDB.

Could you also add e2e tests inflow/e2e/mongo_test.go, similar to Test_Simple_Flow_Partitioned, but for string and numeric _ids for full test coverage.

jgao54 · 2026-06-25T03:49:58Z

+	if minVal >= maxVal {
+		c.logger.Info("[mongo] string min/max range is non-positive, falling back to full table partition")
+		return fullTablePartitions(), nil
+	}


this comparison uses golang instead of db collation, so is prone to false-positive. I think we can trust the db-fetched min/max using its ordering, and just check for equality here.

jgao54 · 2026-06-25T03:55:39Z

+	// boundaries are well distributed even with clustered keys.
+	stringSampleOversample = 20
+	// stringSampleMaxSize caps sampling cost on very large collections.
+	stringSampleMaxSize = 100000


today we limit numPartitions to 1000, so technically the max possible sample size is 20*1000=20000; we'll never hit this limit, but fine to keep as an upper bound

- Add end_inclusive to StringPartitionRange proto (aligns with MySQL PR PeerDB-io#4482 adding the same field; avoids future merge conflict) - Replace sentinel-string approach with end_inclusive=true on the last string partition, using $lte instead of $lt for its MongoDB filter - Sort string _id samples via MongoDB $sort stage instead of Go sort, preserving the collection's collation order - Filter boundary samples by exact equality instead of <=/>= comparison, trusting MongoDB's min/max guarantees - Add e2e tests for string and numeric _id partitioned snapshot flows - Document the double _id gap limitation on numericPartitions (follow-up)

- Add utils.FullTablePartition() helper and use it in place of the local fullTablePartitions() in qrep_partition.go and the inline construction in qrep.go - Remove slices.Compact() from computeStringBoundaries: the seen map already deduplicates interior values, and the quantile index formula is strictly increasing when len(interior) >= numPartitions, so no consecutive duplicates can occur in the else branch - Tighten e2e partition count assertions to exact counts (10) instead of >1 for string and numeric _id tests - Delete Test_Snapshot_Non_ObjectID_Falls_Back_To_Single_Partition: string _id collections are now partitioned rather than falling back to full-table, so the test expectation no longer holds

andreyzhelnin-st · 2026-06-26T15:19:05Z

@jgao54 Addressed comments & rebased
Also go test -count=1 -v -timeout 30m -run 'TestMongoClickhouseSuite' ./e2e/ passes on my laptop.

github-actions · 2026-07-02T02:52:21Z

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

Follow-ups for MongoDB parallel snapshotting #4487 to introduce NumericPartitions; this avoids missing data during initial snapshots when there are interior fractional _ids. A few other smaller clean-up, with inline comments.

andreyzhelnin-st requested a review from a team as a code owner June 24, 2026 15:40

andreyzhelnin-st requested a deployment to external-contributor June 24, 2026 15:40 — with GitHub Actions Waiting

andreyzhelnin-st had a problem deploying to external-contributor June 24, 2026 16:01 — with GitHub Actions Failure

andreyzhelnin-st requested a deployment to external-contributor June 24, 2026 16:01 — with GitHub Actions Waiting

ilidemi reviewed Jun 24, 2026

View reviewed changes

Comment thread flow/connectors/clickhouse/cdc.go

ilidemi requested a review from jgao54 June 24, 2026 21:04

jgao54 reviewed Jun 25, 2026

View reviewed changes

Comment thread flow/connectors/mongo/qrep_partition.go

jgao54 reviewed Jun 25, 2026

View reviewed changes

andreyzhelnin-st force-pushed the mongo-id-partitioning branch from acc2b74 to 4ff837b Compare June 25, 2026 07:32

andreyzhelnin-st requested a deployment to external-contributor June 25, 2026 07:32 — with GitHub Actions Waiting

andreyzhelnin-st requested a deployment to external-contributor June 25, 2026 09:10 — with GitHub Actions Waiting

andreyzhelnin-st added 2 commits June 26, 2026 16:52

mongo: reduce e2e test doc count to 100 for faster local runs

8dac193

andreyzhelnin-st force-pushed the mongo-id-partitioning branch from 7c14f4d to 24e30e2 Compare June 26, 2026 15:15

andreyzhelnin-st requested a deployment to external-contributor June 26, 2026 15:15 — with GitHub Actions Waiting

andreyzhelnin-st force-pushed the mongo-id-partitioning branch from 24e30e2 to b89e2fc Compare June 26, 2026 15:17

andreyzhelnin-st had a problem deploying to external-contributor June 26, 2026 15:17 — with GitHub Actions Failure

andreyzhelnin-st requested a review from jgao54 June 26, 2026 15:18

fix(lint): restore stdlib import group separator in qrep_partition.go

c21475e

andreyzhelnin-st had a problem deploying to external-contributor June 26, 2026 15:47 — with GitHub Actions Failure

Merge remote-tracking branch 'origin/main' into mongo-id-partitioning

1a11c54

ilidemi temporarily deployed to external-contributor July 2, 2026 02:38 — with GitHub Actions Inactive

ilidemi deployed to external-contributor July 2, 2026 02:38 — with GitHub Actions Active

ilidemi temporarily deployed to external-contributor July 2, 2026 02:38 — with GitHub Actions Inactive

jgao54 approved these changes Jul 2, 2026

View reviewed changes

jgao54 merged commit 1e7a9a9 into PeerDB-io:main Jul 2, 2026
15 checks passed

jgao54 mentioned this pull request Jul 3, 2026

mongoDB parallel snapshot follow-up #4531

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

mongo: partition snapshots for numeric and string _id collections#4487

mongo: partition snapshots for numeric and string _id collections#4487
jgao54 merged 6 commits into
PeerDB-io:mainfrom
andreyzhelnin-st:mongo-id-partitioning

andreyzhelnin-st commented Jun 24, 2026

Uh oh!

CLAassistant commented Jun 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

jgao54 Jun 25, 2026 •

edited

Loading

Uh oh!

jgao54 Jun 25, 2026 •

edited

Loading

Uh oh!

jgao54 left a comment

Uh oh!

jgao54 Jun 25, 2026

Uh oh!

jgao54 Jun 25, 2026

Uh oh!

andreyzhelnin-st commented Jun 26, 2026

Uh oh!

github-actions Bot commented Jul 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

andreyzhelnin-st commented Jun 24, 2026

Summary

Approach

Changes

Testing

Notes / limitations

Uh oh!

CLAassistant commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jgao54 Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jgao54 Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jgao54 left a comment

Choose a reason for hiding this comment

Uh oh!

jgao54 Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

jgao54 Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

andreyzhelnin-st commented Jun 26, 2026

Uh oh!

github-actions Bot commented Jul 2, 2026

Code review

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CLAassistant commented Jun 24, 2026 •

edited

Loading

jgao54 Jun 25, 2026 •

edited

Loading

jgao54 Jun 25, 2026 •

edited

Loading